Media synchronized control of peripherals

ABSTRACT

A computer system and method for delivering synchronized metadata for a media to a target device that is associated with a plurality of peripherals.

DESCRIPTION OF RELATED ART

Media producers derive significant revenue from associated merchandise,e.g. toys such as Buzz Lightyear doll produced by Disney as a companionto Toy Story. Some of these types of toys contain sophisticatedelectronics that allow the toy to be controlled externally, e.g. viavoice or via data sent to the toy over a wireless network.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments are illustrated in the figures of the accompanying drawings,which are meant to be exemplary and not limiting, and in which likereferences are intended to refer to like or corresponding things.

FIG. 1 is a system diagram of an environment in which at least one ofthe various embodiments may be implemented.

FIG. 2 depicts exemplary representations for baseline synchronizedmetadata.

FIG. 3 shows an exemplary flow showing binary control data targetedbased on peripherals.

FIG. 4 depicts exemplary representations for baseline system enhancedwith binary data packets intended for peripherals.

FIG. 5 illustrates an example of metadata packet intended for aperipheral device.

FIG. 6 depicts an overall architecture and the data flow of the metadatadelivery system.

FIG. 7 illustrates another example of metadata packet intended for aperipheral device.

FIG. 8 illustrates an example of possible implementation of actionsincluded in a Response Handler.

FIG. 9 depicts the data flow of a metadata delivery system comprising aperipheral device.

FIG. 10 depicts the data flow of a metadata delivery system definingstandard interfaces for the peripheral device and Response Handleractions.

FIG. 11 shows a flow chart for processing a packet on a target device.

FIG. 12 shows a flow chart for executing Response Handler on the targetdevice.

FIG. 13 depicts the data flow of a metadata delivery system that createsGUI adjustment triggers.

FIG. 14 depicts the data flow of processing audio & video packets froman elementary stream in a media source.

FIG. 15 illustrates an example of finding relationships between films toestimate GUI triggers.

FIG. 16 illustrates an example of segments of a data map.

FIG. 17 illustrates an example of segments of a data map with GUItriggers.

FIG. 18 shows an embodiment of a target device that may be included in asystem such as that shown in FIG. 1;

FIG. 19 shows an embodiment of a network computer that may be includedin a system such as that shown in FIG. 1;

FIG. 20 illustrates a plurality of databases that may be included in asystem such as that shown in FIG. 1.

DETAILED DESCRIPTION OF THE EMBODIMENTS

It is to be understood that the figures and descriptions of the presentinvention have been simplified to illustrate elements that are relevantfor a clear understanding of the present invention, while eliminating,for purposes of clarity, many other elements which are conventional inthis art. Those of ordinary skill in the art will recognize that otherelements are desirable for implementing the present invention. However,because such elements are well known in the art, and because they do notfacilitate a better understanding of the present invention, a discussionof such elements is not provided herein.

The use of the terms “a,” “an,” “at least one,” “one or more,” andsimilar terms indicate one of a feature or element as well as more thanone of a feature. The use of the term “the” to refer to the feature doesnot imply only one of the feature and element.

When an ordinal number (such as “first,” “second,” “third,” and so on)is used as an adjective before a term, that ordinal number is used(unless expressly or clearly specified otherwise) merely to indicate aparticular feature, such as to distinguish that particular feature fromanother feature that is described by the same term or by a similar term.

When a single device, article or other product is described herein, morethan one device/article (whether or not they cooperate) mayalternatively be used in place of the single device/article that isdescribed. Accordingly, the functionality that is described as beingpossessed by a device may alternatively be possessed by more than onedevice/article (whether or not they cooperate). Similarly, where morethan one device, article or other product is described herein (whetheror not they cooperate), a single device/article may alternatively beused in place of the more than one device or article that is described.Accordingly, the various functionality that is described as beingpossessed by more than one device or article may alternatively bepossessed by a single device/article.

The functionality and/or the features of a single device that isdescribed may be alternatively embodied by one or more other devices,which are described but are not explicitly described as having suchfunctionality/features. Thus, other embodiments need not include thedescribed device itself, but rather can include the one or more otherdevices that would, in those other embodiments, have suchfunctionality/features.

Various embodiments will now be described in detail on the basis ofexemplary embodiments. Embodiments disclosed herein may be practicedusing programmable digital computers and networks therefor.

Disclosed is a system that is able to deliver specialized data packets,such as metadata packets, to a target device for a particular piece ofmedia.

FIG. 1 shows components of one embodiment of an environment in whichembodiments of the innovations described herein may be practiced. Notall of the components may be required to practice the innovations, andvariations in the arrangement and type of the components may be madewithout departing from the spirit or scope of the innovations. As shown,system 100 of FIG. 1 includes a media device or system 107; local areanetworks (LANs) wide area networks (WANs)—(network) 110; wirelessnetwork 108; target devices 101(a), 101(b) . . . 101(n), which may be atablet, a set top box or a BD/DVD player; metadata database 103;metadata delivery system 104 that delivers metadata stored in metadatadatabase 103 to the target device 101(a), 101(b) . . . 101(n); andperipheral devices 102(a), 102(b) . . . 102(n), such as toys orsecondary display device, which can be controlled remotely, for example,over a wireless network. The target device 101(a), 101(b) . . . 101(n)and the peripheral devices 102(a), 102(b) . . . 102(n) exchangeinformation via a discovery process during which identification data isexchanged. Exemplary metadata delivery systems are described in U.S.patent application Ser. No. 14/588,143 entitled Management,Categorization, Contextualizing and Sharing of Metadata-Based Contentand U.S. application Ser. No. 14/587,926, entitled Identifying andCategorizing Contextual Data for Media, and U.S. patent application Ser.No. 14/588,224 entitled Lull Management for Content Delivery, each ofwhich were filed on Dec. 31, 2014 concurrently herewith by the sameApplicant, OpenTV, Inc., and the entirety each of which is incorporatedby reference herein.

At least one embodiment of target devices 101(a)-101(n) is described inmore detail below in conjunction with FIG. 18. In one embodiment, atleast some of target devices 101(a)-101(n) may operate over a wiredand/or wireless network, such as networks 110 and/or 108. Generally,target devices 101(a)-101(n) may include virtually any computer capableof communicating over a network to send and receive information, performvarious online activities, offline actions, or the like. In variousembodiments, one or more of target devices 101(a)-101(n) can beconfigured to operate with conjunction with a media device or system107, for example a television, radio, another computer, a tablet device,as smart phone, or any device enabled to allow a user to consume media.For example, target devices 101(a)-101(n) can be configured to send datafrom a user consuming a media product on a media device or system 107,for example a movie, television show, or listening to music from oranother source or media device or system, e.g.: a television, a radio, ahome theater, a sound system, another computer, or even in a movietheater, etc. In various embodiments the media product may be providedto the user locally, for example via DVD, CD, or any locally orphysically stored media, etc. and can also be provided by a mediacontent delivery service, such as a streaming media service, which canbe provided by the metadata delivery system 104 to the other mediadevice (e.g. a television or another computer). Target devices101(a)-101(n) need not of course be constrained to such use and may alsobe employed, for example, as an end-user computing node, or any othertypical use for a computer device.

Computers that may operate as target device 101(a) may include computersthat typically connect using a wired or wireless communications mediumsuch as personal computers, multiprocessor systems, microprocessor-basedor programmable electronic devices, network PCs, or the like. In someembodiments, target devices 101(a)-101(n) may include virtually anyportable personal computer capable of connecting to another computingdevice and receiving information. However, portable computers are not solimited and may also include other portable devices such as cellulartelephones, radio frequency (RF) devices, infrared (IR) devices,Personal Digital Assistants (PDAs), handheld computers, wearablecomputers, integrated devices combining one or more of the precedingdevices, and the like. As such, target devices 101(a)-101(n) typicallyrange widely in terms of capabilities and features. Moreover, targetdevices 101(a)-101(n) may access various computing applications,including a browser, or other web-based application.

A web-enabled target device may include a browser application that isconfigured to receive and to send web pages, web-based messages, and thelike. The browser application may be configured to receive and displaygraphics, text, multimedia, and the like, employing virtually anyweb-based language, including a wireless application protocol messages(WAP), and the like. In one embodiment, the browser application isenabled to employ Handheld Device Markup Language (HDML), WirelessMarkup Language (WML), WMLScript, JavaScript, Standard GeneralizedMarkup Language (SGML), HyperText Markup Language (HTML), eXtensibleMarkup Language (XML), and the like, to display and send a message. Inone embodiment, a user of the target device may employ the browserapplication to perform various activities over a network (online).However, another application may also be used to perform various onlineactivities.

Target devices 101(a)-101(n) may also include at least one other targetdevice application that is configured to receive and/or send contentbetween another computer. The target device application may include acapability to send and/or receive content, or the like. The targetdevice application may further provide information that identifiesitself, including a type, capability, name, and the like. In oneembodiment, target devices 101(a)-101(n) may uniquely identifythemselves through any of a variety of mechanisms, including an InternetProtocol (IP) address, a phone number, Mobile Identification Number(MIN), an electronic serial number (ESN), or other device identifier.Such information may be provided in a network packet, or the like, sentbetween other target devices, metadata delivery system 104, or othercomputers.

Target devices 101(a)-101(n) may further be configured to include atarget device application that enables an end-user to log into anend-user account that may be managed by another computer, such asmetadata delivery system 104, or the like. Such end-user account, in onenon-limiting example, may be configured to enable the end-user to manageone or more online activities, including in one non-limiting example,search activities, social networking activities, browse variouswebsites, communicate with other users, or the like. However,participation in such online activities may also be performed withoutlogging into the end-user account.

Wireless network 108 is configured to couple target devices101(b)-101(n) and its components with network 110. Wireless network 108may include any of a variety of wireless sub-networks that may furtheroverlay stand-alone ad-hoc networks, and the like, to provide aninfrastructure-oriented connection for target devices 101(b)-101(n).Such sub-networks may include mesh networks, Wireless LAN (WLAN)networks, cellular networks, and the like. In one embodiment, the systemmay include more than one wireless network.

Wireless network 108 may further include an autonomous system ofterminals, gateways, routers, and the like connected by wireless radiolinks, and the like. These connectors may be configured to move freelyand randomly and organize themselves arbitrarily, such that the topologyof wireless network 108 may change rapidly.

Wireless network 108 may further employ a plurality of accesstechnologies including 2nd (2G), 3rd (3G), 4th (4G) 5th (5G) generationradio access for cellular systems, WLAN, Wireless Router (WR) mesh, andthe like. Access technologies such as 2G, 3G, LTE, 4G, 5G, and futureaccess networks may enable wide area coverage for mobile devices, suchas target devices 101(b)-101(n ) with various degrees of mobility. Inone non-limiting example, wireless network 108 may enable a radioconnection through a radio network access such as Global System forMobile communication (GSM), General Packet Radio Services (GPRS),Enhanced Data GSM Environment (EDGE), code division multiple access(CDMA), time division multiple access (TDMA), Wideband Code DivisionMultiple Access (WCDMA), High Speed Downlink Packet Access (HSDPA), LongTerm Evolution (LTE), and the like. In essence, wireless network 108 mayinclude virtually any wireless communication mechanism by whichinformation may travel between target devices 101(b)-101(n) and anothercomputer, network, and the like.

Network 110 is configured to couple network computers with othercomputers and/or computing devices, including, metadata delivery system104, target device 101(a), and target devices 101(b)-101(n) throughwireless network 108. Network 110 is enabled to employ any form ofcomputer readable media for communicating information from oneelectronic device to another. Also, network 110 can include the Internetin addition to local area networks (LANs), wide area networks (WANs),direct connections, such as through a universal serial bus (USB) port,other forms of computer-readable media, or any combination thereof. Onan interconnected set of LANs, including those based on differingarchitectures and protocols, a router acts as a link between LANs,enabling messages to be sent from one to another. In addition,communication links within LANs typically include twisted wire pair orcoaxial cable, while communication links between networks may utilizeanalog telephone lines, full or fractional dedicated digital linesincluding T1, T2, T3, and T4, and/or other carrier mechanisms including,for example, E-carriers, Integrated Services Digital Networks (ISDNs),Digital Subscriber Lines (DSLs), wireless links including satellitelinks, or other communications links known to those skilled in the art.Moreover, communication links may further employ any of a variety ofdigital signaling technologies, including without limit, for example,DS-0, DS-1, DS-2, DS-3, DS-4, OC-3, OC-12, OC-48, or the like.Furthermore, remote computers and other related electronic devices couldbe remotely connected to either LANs or WANs via a modem and temporarytelephone link. In one embodiment, network 110 may be configured totransport information of an Internet Protocol (IP). In essence, network110 includes any communication method by which information may travelbetween computing devices.

Additionally, communication media typically embodies computer readableinstructions, data structures, program modules, or other transportmechanism and includes any information delivery media. By way ofexample, communication media includes wired media such as twisted pair,coaxial cable, fiber optics, wave guides, and other wired media andwireless media such as acoustic, RF, infrared, and other wireless media.

One embodiment of a metadata delivery system 104 is described in moredetail below in conjunctions with FIG. 19. Briefly, however, metadatadelivery system 104 includes virtually any network computer capable ofdelivering metadata-based content to a client user and acceptingrequests and data therefrom. For example, target devices 101(a)-101(n)can be configured to send data from a user consuming a media product,for example a movie, television show, or listening to music from oranother source or media device, e.g.: a television, a radio, a, movietheater, etc. The metadata delivery system 104 can then deliver, interalia, complementary synchronous metadata-based content based on theidentification of the media being consumed by the user. Computers thatmay be arranged to operate as metadata delivery system 104 includevarious network computers, including, but not limited to multiprocessorsystems, server computers, and the like.

Although the metadata delivery system 104 may be a single computer, theinvention is not so limited. For example, one or more functions of themetadata delivery system 104 may be distributed across one or moredistinct network computers. Moreover, metadata delivery system 104 isnot limited to a particular configuration. Thus, in one embodiment,metadata delivery system 104 may contain a plurality of networkcomputers. In another embodiment, metadata delivery system 104 maycontain a plurality of network computers that operate using amaster/slave approach, where one of the plurality of network computersof metadata delivery system 104 is operative to manage and/or otherwisecoordinate operations of the other network computers. In otherembodiments, the metadata delivery system 104 may operate as a pluralityof network computers arranged in a cluster architecture, a peer-to-peerarchitecture, and/or even within a cloud architecture. Thus, embodimentsare not to be construed as being limited to a single environment, andother configurations, and architectures are also envisaged.

Although illustrated separately, metadata database 103 and metadatadelivery system 104 may be employed as a single network computer,separate network computers, a cluster of network computers, or the like.In some embodiments, either metadata database 103 or metadata deliverysystem 104, or both, may be enabled to deliver content, respond to userinteractions with the content, track user interaction with the content,update widgets and widgets controllers, or the like.

FIG. 18 shows one embodiment of Target Device 2200 that may be includedin a system implementing embodiments of the invention. Target Device2200 may include many more or less components than those shown in FIG.18. However, the components shown are sufficient to disclose anillustrative embodiment for practicing the present invention. TargetDevice 2200 may represent, for example, one embodiment of at least oneof Target Device 2200 101(a)-101(n) of FIG. 1.

As shown in the figure, Target Device 2200 includes a processor 2202 incommunication with a mass memory 2226 via a bus 2234. In someembodiments, processor 2202 may include one or more central processingunits (CPU). Target Device 2200 also includes a power supply 2228, oneor more network interfaces 2236, an audio interface 2238, a display2240, a keypad 2242, an illuminator 2244, a video interface 2246, aninput/output interface 2248, a haptic interface 2250, and a globalpositioning system (GPS) receiver 2232 or other geolocation components,Power supply 2228 provides power to Target Device 2200. A rechargeableor non-rechargeable battery may be used to provide power. The power mayalso be provided by an external power source, such as an alternatingcurrent (AC) adapter or a powered docking cradle that supplements and/orrecharges a battery.

Target Device 2200 may optionally communicate with a base station (notshown), or directly with another computer. Network interface 2236includes circuitry for coupling Target Device 2200 to one or morenetworks, and is constructed for use with one or more communicationprotocols and technologies including, but not limited to, GSM, CDMA,TDMA, GPRS, EDGE, WCDMA, HSDPA, LTE, user datagram protocol (UDP),transmission control protocol/Internet protocol (TCP/IP), short messageservice (SMS), WAP, ultra wide band (UWB), IEEE 802.16 WorldwideInteroperability for Microwave Access (WiMax), session initiatedprotocol/real-time transport protocol (SIP/RTP), or any of a variety ofother wireless communication protocols. Network interface 2236 issometimes known as a transceiver, transceiving device, or networkinterface card (NIC).

Audio interface 2238 is arranged to produce and receive audio signalssuch as the sound of media from another source (e.g., television, radio,etc.). For example, audio interface 2238 may be coupled to a speaker andmicrophone (not shown) to enable telecommunication with others and/orgenerate an audio acknowledgement for some action.

Display 2240 may be a liquid crystal display (LCD), gas plasma, lightemitting diode (LED), organic LED, or any other type of display usedwith a computer. Display 2240 may also include a touch sensitive screenarranged to receive input from an object such as a stylus or a digitfrom a human hand.

Keypad 2242 may comprise any input device arranged to receive input froma user. For example, keypad 2242 may include a push button numeric dial,or a keyboard. Keypad 2242 may also include command buttons that areassociated with selecting and sending images.

Illuminator 2244 may provide a status indication and/or provide light.Illuminator 2244 may remain active for specific periods of time or inresponse to events. For example, when illuminator 2244 is active, it maybacklight the buttons on keypad 2242 and stay on while the Target Deviceis powered. Also, illuminator 2244 may backlight these buttons invarious patterns when particular actions are performed, such as dialinganother target device. Illuminator 2244 may also cause light sourcespositioned within a transparent or translucent case of the target deviceto illuminate in response to actions.

Video interface 2246 is arranged to capture video images, such as astill photo, a video segment, an infrared video, or the like. Forexample, video interface 2246 may be coupled to a digital video camera,a web-camera, or the like. Video interface 2246 may comprise a lens, animage sensor, and other electronics. Image sensors may include acomplementary metal-oxide-semiconductor (CMOS) integrated circuit,charge-coupled device (CCD), or any other integrated circuit for sensinglight.

Target Device 2200 also comprises input/output interface 2248 forcommunicating with external devices, such as a headset, or other inputor output devices not shown in FIG. 18. Input/output interface 2248 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

Haptic interface 2250 is arranged to provide tactile feedback to a userof the target device. For example, the haptic interface 2250 may beemployed to vibrate target device 2200 in a particular way when anotheruser of a computing computer is calling. Accelerometers and otherkinetic or force-based interfaces can be included as well.

Target device 2200 may also include GPS transceiver 2232 to determinethe physical coordinates of target device 2200 on the surface of theEarth. GPS transceiver 2232, in some embodiments, may be optional. GPStransceiver 2232 typically outputs a location as latitude and longitudevalues. However, GPS transceiver 2232 can also employ othergeo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), Enhanced Observed Time Difference(E-OTD), Cell Identifier (CI), Service Area Identifier (SAI), EnhancedTiming Advance (ETA), Base Station Subsystem (BSS), or the like, tofurther determine the physical location of target device 2200 on thesurface of the Earth. It is understood that under different conditions,GPS transceiver 2232 can determine a physical location withinmillimeters for target device 2200; and in other cases, the determinedphysical location may be less precise, such as within a meter orsignificantly greater distances. In one embodiment, however, targetdevice 2200 may through other components, provide other information thatmay be employed to determine a physical location of the computer,including for example, a Media Access Control (MAC) address, IP address,or the like.

Mass memory 2226 includes a Random Access Memory (RAM) 2204, a Read-onlyMemory (ROM) 2222, and other storage means. Mass memory 2226 illustratesan example of computer readable storage media (devices) for storage ofinformation such as computer readable instructions, data structures,program modules or other data. Mass memory 2226 stores a basicinput/output system (BIOS) 2224 for controlling low-level operation oftarget device 2200. The mass memory also stores an operating system 2206for controlling the operation of target device 2200. It will beappreciated that this component may include a general-purpose operatingsystem such as a version of UNIX, or LINUX™, or a specialized clientcommunication operating system such as Microsoft Corporation's WindowsMobile™, Apple Corporation's iOS™, Google Corporation's Android™ or theSymbian® operating system. The operating system may include, orinterface with a Java virtual machine module that enables control ofhardware components and/or operating system operations via Javaapplication programs.

Mass memory 2226 further includes one or more data storage 2208, whichcan be utilized by target device 2200 to store, among other things,applications 2214 and/or other data. For example, data storage 2208 mayalso be employed to store information that describes variouscapabilities of target device 2200. The information may then be providedto another computer based on any of a variety of events, including beingsent as part of a header during a communication, sent upon request, orthe like. Data storage 2208 may also be employed to store socialnetworking information including address books, buddy lists, aliases,user profile information, or the like. Further, data storage 2208 mayalso store message, web page content, or any of a variety of usergenerated content. At least a portion of the information may also bestored on another component of target device 2200, including, but notlimited to processor readable storage media 2230, a disk drive or othercomputer readable storage devices (not shown) within target device 2200.

Processor readable storage media 2230 may include volatile, nonvolatile,removable, and non-removable media implemented in any method ortechnology for storage of information, such as computer- orprocessor-readable instructions, data structures, program modules, orother data. Examples of computer readable storage media include RAM,ROM, Electrically Erasable Programmable Read-only Memory (EEPROM), flashmemory or other memory technology, Compact Disc Read-only Memory(CD-ROM), digital versatile disks (DVD) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other physical medium which can be usedto store the desired information and which can be accessed by acomputer. Processor readable storage media 2230 may also be referred toherein as computer readable storage media and/or computer readablestorage device.

Applications 2214 may include computer executable instructions which,when executed by target device 2200, transmit, receive, and/or otherwiseprocess network data. Network data may include, but is not limited to,messages (e.g. SMS, Multimedia Message Service (MMS), instant message(IM), email, and/or other messages), audio, video, and enabletelecommunication with another user of another client computer.Applications 2214 may include, for example, browser 2218, and otherapplications 2220 including those described with respect to FIG. 9.Other applications 2220 may also include, but are not limited to,calendars, search programs, email clients, IM applications, SMSapplications, voice over Internet Protocol (VOIP) applications, contactmanagers, task managers, transcoders, database programs, word processingprograms, security applications, spreadsheet programs, games, searchprograms, and so forth.

Browser 2218 may include virtually any application configured to receiveand display graphics, text, multimedia, messages, and the like,employing virtually any web based language. In one embodiment, thebrowser application is enabled to employ HDML, WML, WMLScript,JavaScript, HTML, XML, and the like, to display and send a message.However, any of a variety of other web-based programming languages maybe employed. In one embodiment, browser 2218 may enable a user of targetdevice 2200 to communicate with another network computer, such asMetadata Delivery System 104 of FIG. 1.

Applications 2214 may also include Widget Controller 2210 and one ormore Widgets 2212. Widgets 2212 may be collections of content providedto the target device by Metadata Delivery System 104. Widget Controller2210 may be a program that may be provided to the target device byMetadata Delivery System 104. Widget Controller 2210 and Widgets 2212may run as native target device applications or they may run in Browser2218 as web browser based applications. Also, Widget Controller 2210 andWidgets 2212 may be arranged to run as native applications or webbrowser applications, or combination thereof.

As used herein, the term “widget controller” refers to a computerprogram that may be operative on a client application. Widgetcontrollers may be downloaded and/or otherwise deployed to a clientapplication. Widget controllers may be arranged to be operative fordownloading content, monitoring media, or otherwise managing widgetslocated within client applications.

As used herein, the term “widget” refers to a user-interface elementlocated in the client application. Widgets may be invisible or visibleto users of the client applications. In some cases, a widget controllermay generate widget “on-the-fly” before deploying content into thewidget. Widgets may be adapted to reflect the operating environment ofthe client application that they are being hosted within. For example,in clients that support HTML, CSS a widget may be an HTML element suchas a DIV, P, or the like. For client application operative in a Javaenvironment, a widget may be a View object or Window object, and so on.

FIG. 19 shows one embodiment of a network computer 2300, according toone embodiment of the invention. Network computer 2300 may include manymore or less components than those shown. The components shown, however,are sufficient to disclose an illustrative embodiment for practicing theinvention. Network computer 2300 may be configured to operate as aserver, client, peer, a host, or any other computer. Network computer2300 may represent, for example Metadata Delivery System 104 of FIG. 1,and/or other network computers.

Network computer 2300 includes processor 2302, processor readablestorage media 2328, network interface unit 2330, an input/outputinterface 2332, hard disk drive 2334, video display adapter 2336, andmemory 2326, all in communication with each other via bus 2338. In someembodiments, processor 2302 may include one or more central processingunits.

As illustrated in FIG. 19, network computer 2300 also can communicatewith the Internet, or some other communications network, via networkinterface unit 2330, which is constructed for use with variouscommunication protocols including the TCP/IP protocol. Network interfaceunit 2330 is sometimes known as a transceiver, transceiving device, ornetwork interface card (NIC).

Network computer 2300 also comprises input/output interface 2332 forcommunicating with external devices, such as a keyboard, or other inputor output devices not shown in FIG. 19. Input/output interface 2332 canutilize one or more communication technologies, such as USB, infrared,Bluetooth™, or the like.

Memory 2326 generally includes RAM 2304, ROM 2322 and one or morepermanent mass storage devices, such as hard disk drive 2334, tapedrive, optical drive, and/or floppy disk drive. Memory 2326 storesoperating system 2306 for controlling the operation of network computer2300. Any general-purpose operating system may be employed. Basicinput/output system (BIOS) 2324 is also provided for controlling thelow-level operation of network computer 2300.

Although illustrated separately, memory 2326 may include processorreadable storage media 2328. Processor readable storage media 2328 maybe referred to and/or include computer readable media, computer readablestorage media, and/or processor readable storage device. Processorreadable storage media 2328 may include volatile, nonvolatile,removable, and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data. Examplesof processor readable storage media include RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other media whichcan be used to store the desired information and which can be accessedby a computer.

Memory 2326 further includes one or more data storage 2308, which can beutilized by network computer 2300 to store, among other things,applications 2314 and/or other data such as content 2310. For example,data storage 2308 may also be employed to store information thatdescribes various capabilities of network computer 2300. The informationmay then be provided to another computer based on any of a variety ofevents, including being sent as part of a header during a communication,sent upon request, or the like. Data storage 2308 may also be employedto store messages, web page content, or the like. At least a portion ofthe information may also be stored on another component of networkcomputer 2300, including, but not limited to processor readable storagemedia 2328, hard disk drive 2334, or other computer readable storagemedias (not shown) within network compute 2300.

Data storage 2308 may include a database, text, spreadsheet, folder,file, or the like, that may be configured to maintain and store useraccount identifiers, user profiles, email addresses, IM addresses,and/or other network addresses; or the like.

In at least one of the various embodiments, data storage 2308 mayinclude databases 2310. In various embodiments, as shown in FIG. 20,databases include such as one or more signature databases (e.g. an audiosignature database 2401, a video signature database 2402, a narrativesignature database 2403), a segment database 2404, an associationsdatabase 2405, a metadata database 2406, a media playlist database 2407,a context database 2409, and a context category database 2410. Althoughthe system shows the databases 2310 as included in the system and server2300, one or more of the databases can be external to the server orsystem and operatively connected thereto.

Returning to FIG. 19, data storage 2308 may further include programcode, data, algorithms, and the like, for use by a processor, such asprocessor 2302 to execute and perform actions. In one embodiment, atleast some of data store 2308 might also be stored on another componentof network computer 2300, including, but not limited toprocessor-readable storage media 2328, hard disk drive 2334, or thelike.

Applications 2312 may include computer executable instructions, whichmay be loaded into mass memory and run on operating system 2306.Examples of application programs may include transcoders, schedulers,calendars, database programs, word processing programs, HypertextTransfer Protocol (HTTP) programs, customizable user interface programs,IPSec applications, encryption programs, security programs, SMS messageservers, IM message servers, email servers, account managers, and soforth. Applications 2312 may also include, for example, a website server2314, a Common Segment Detector Application 2316, one or more ContextDifference Detector Applications 2318, a Data Categorizer Application2319, a Context Assignment Module 2320, and/or Context Filtering andRequest Handler 2321.

Website server 2314 may represents any of a variety of information andservices that are configured to provide content, including messages,over a network to another computer. Thus, website server 2314 caninclude, for example, a web server, a File Transfer Protocol (FTP)server, a database server, a content server, or the like. Website server2314 may provide the content including messages over the network usingany of a variety of formats including, but not limited to HTML, XML,Compact HTML (cHTML), Extensible HTML (xHTML), or the like.

In at least one of the various embodiments, Applications 2312 may beoperative on Metadata Delivery System 104 of FIG. 1. Applications 2312may employ processes, or parts of processes, similar to those describedin conjunction with FIGS. 2-17, to perform at least some of its actions.

FIG. 2 depicts an example of representations for baseline synchronizedmetadata. As shown in FIG. 2, the synchronized textual metadata 201(a),201(b) . . . 201(n) contains precise timing information related to theA/V stream timeline 203 of media 202 being played back, which results inthat the target device is able to synchronize the rendering of themetadata to the playback. For example, when the media 202 is movie ToyStory, the synchronized textual metadata 201(a) indicates that the typeof metadata 201(a) is fact, and at time point “1874450” text “Toy Storyis a 1995 American computer-animated buddy-comedy adventure filmproduced by Pixar Animation Studios and released by Walt DisneyPictures. Directed by John Lasseter, Toy Story was the firstfeature-length computer-animated film and the first theatrical filmproduced by Pixar” will be displayed on the screen associated with thetarget device.

FIG. 3 shows an exemplary flow showing binary control data targetedbased on peripherals. As shown in FIG. 3, target devices 301(a) and301(b) are connected to peripheral devices 302(a), 302(b), 302(c), and302(d), respectively. Each type of peripheral device has its own set ofcontrol commands that are unique for that type of peripheral. Sequencesof control commands, which can have very precise timing, are configuredto control the peripheral (e.g., toy) to perform different functions.

As shown in FIG. 3, metadata database 303 provides metadata to themetadata delivery system 304. Target device 301(a) communicates theperipheral identification information 305, which is data about theconnected peripheral devices 302(a) and 302(b), to the metadata deliverysystem 304 so that the metadata delivery system 304 may send peripheralcontrol data 307, which is only relevant for peripheral device R 302(a)and peripheral device S 302(b), to target device 301(a). Similarly,target device 301(b) communicates the peripheral identificationinformation 306, which is data about the connected peripheral devices302(b) and 302(c), to the metadata delivery system 304 so that themetadata delivery system 304 may send peripheral control data 308, whichis only relevant for peripheral device S 302(b) and peripheral device T302(c), to target device 301(b). It will be noted that a peripheraldevice may be connected to one or more target devices. The controlcommands for the peripherals are inserted into binary metadata packetsand distributed to the target device using the metadata delivery system,which will be described below referring to FIG. 4.

FIG. 4 depicts exemplary representations for baseline system enhancedwith binary data packets intended for peripherals. As shown in FIG. 4,in addition to the synchronized textual metadata 201(a)-201(e) thatcorresponds to the A/V stream timeline 203 of the media 202 which hasbeen described in FIG. 2, synchronized binary metadata 404(a)-404(f)that corresponds to the A/V stream timeline 203 is also delivered to thetarget device. The binary data packets include a textual header withsome peripheral specific addressing information in text format and abinary payload holding the control data for the peripheral. An exampleof the binary data packet 404(a) will be described below referring toFIG. 5.

FIG. 5 illustrates an example of metadata packet intended for aperipheral device. In the metadata packet 404(a), type 501“PeripheralData” indicates that metadata packet 404(a) is intended for aperipheral device. Time 502 indicates the time point, for example,1874472, at which the intended peripheral device should perform aparticular action designated by the metadata packet 404(a). PeripheralID 503 indicates that metadata packet 404(a) is intended for aparticular peripheral device with ID “Disney_35”. Payload 504 indicatesthe action indicated by “0xAB01F12C . . . ” to be performed by theparticular peripheral device.

Referring back to FIG. 4, for example, synchronized binary metadata404(a) includes information that means that the peripheral device“Jessie doll” with version higher than 2.x should perform action“giggle” at time point 1874472. Similarly, metadata 404(b) means thatthe peripheral device “Woody doll” with version higher than 1.4x shouldperform action “raise right arm” at time point 1874472; metadata 404(c)means that the peripheral device “Woody doll” should speak “Hi Jessie.Long time no see” at time point 1874496; metadata 404(d) means that theperipheral device “Buzz toy” with version no lower than 1.0 should speak“To infinity and beyond” at time point 1874472; metadata 404(e) meansthat the peripheral device “Jessie doll” with version higher than 2.xshould perform action “kick left leg” at time point 1874496; andmetadata 404(f) means that the peripheral device “Buzz toy” with versionno lower than 1.0 should perform action “turn head right and blink” attime point 1874496.

FIG. 6 depicts an exemplary embodiment of an overall architecture anddata flow of the metadata delivery system. As shown in FIG. 6, metadatadelivery system 604 delivers standard textual metadata 606 andperipheral specific binary data 607 to the target device 601. Theperipheral specific binary data 607 includes binary metadata607(a)-607(c), which is intended to peripheral devices 602(a)-602(c),respectively. The target device 601 distributes the binary metadata607(a)-607(c) to the corresponding peripheral devices 602(a)-602(c),which will perform actions according to the distributed metadata.

As shown in FIG. 6, the target device typically renders the deliveredmetadata on its associated screen 605, e.g. a TV for a set top box. Inthe case of binary data packets meant for a peripheral, the packet datamay not be rendered, but instead broadcast to the appropriate peripheralvia a wireless network to which both the target device and peripheralsare connected.

In various embodiments, the sourcing of the binary data packets meantfor the peripherals is external to the metadata delivery system and thepackets can be sourced in a just-in-time fashion before being deliveredto the target device. This type of external sourcing allows forreal-time changes to the peripheral control data, e.g. to fix bugs or tomodify the behavior of the peripheral, e.g. the toy, by the peripheralmaker. One exemplary advantage of external sourcing is that the systemcan be configured to avoid having peripheral control data permanentlycoded into a non-external video delivery stream (e.g. coded or providedin a local device or product) and hence not being changeable over time.For example, Toy Story has been distributed on DVD, BD and in streamingformat. A Toy Story DVD purchased 10 years ago can be played still inany DVD player. That DVD contains no information beyond the movieitself. If Disney produced a wirelessly controllable Woody doll nextyear, none of the millions of existing Toy Story DVD's in circulationwould be able to control the Woody doll and thus, the Woody doll wouldnot be able to interact with the media as it is being played back.

Further, in the metadata delivery system, each piece of data deliveredto the target device carries timing information relative to the mediabeing played back. In this way the target device can buffer data andonly act on the data at exactly the right time, thus achieving precisesynchronization with the playback of the media.

For binary packets intended for peripherals, the packets are broadcastto a relevant, connected peripheral. In certain embodiments, distancesbetween the target device and the peripherals are short and hencenetwork transmission quality can be very good with negligible lagbetween the target device broadcast moment and the reception of the databy the peripheral. This means that the peripheral can derive itsinternal timing in part based on when it receives data and thus, highlyprecise synchronization between the peripheral and the media can beachieved even without a two-way protocol such as RTSP. If the targetdevice loses sync with the media, this can be communicated to theperipherals. In various embodiments the target device can be configuredto stop transmitting control data to the peripherals until the targetdevice has regained synch with the media.

In various embodiments, each peripheral may communicate directly withthe metadata delivery system, so that the flow of data may be managedwithout using a target device.

Accordingly, in various embodiments the system is configured to enablethe peripheral devices to synchronize behaviors independently of the waythe media item is distributed. Additionally, in various embodiments thebehaviors do not have to be prebuilt into a peripheral, for example, ina toy at shipping. Thus the peripheral's behavior can change and evolveover time. Thus the peripheral can be synchronized to a more expansiveand evolving media as the peripheral's behaviors can be updated ormodified for each new media item.

For example, where the peripheral is a toy, the toy ‘recognizes’ a videosequence from a new movie related to the toy, and multiple behaviors canbe supported for any particular video sequence, and not be limited tothe inbuilt memory capacity of the toy; new behaviors can be introducedover time keeping the toy's behavior fresh.

In various embodiments, a peripheral can be configured to learn and addnew behaviors which can then be stored in the system. For example, whenthe user of the toy moves the toy while watching/synchronized to a videosequence, the toy's sensors can record the movements introduced by theuser. When the toy is moved by the user to perform actions of, forinstance, a dance, these actions may be automatically uploaded into thesystem and used by other toys for that particular video sequence. A toycan learn how to dance to a particular video sequence by aggregating howthe toy users' moves their toy during that sequence. The aggregationalgorithm may apply methods that include averaging and smoothing thelimb movements, for example, to effectively crowd source a new toy danceor movements for that video sequence.

As shown in FIG. 6, in various embodiments the peripheral devices havetheir own class of metadata packets, and do not generate their ownresponse data. In other embodiments, the peripheral devices can generatetheir own data to be sent to other devices, such as the target device.Thus in various embodiments, there can be a class of metadata packetsintended for peripheral device that generates response data, which isdescribed below referring to FIG. 7.

FIG. 7 illustrates an example of metadata packet intended for aperipheral device that is capable of generating its own response dataasynchronously and sending the response data back to the target device.The target device can monitor the motions of the peripheral device byreading at the response data sent back from the peripheral device. Themetadata packets include an identifier indicating the peripheral cansend response data and also include a Response Handler portion, afterthe Peripheral Payload portion.

In the metadata packet 700, type 701 indicates that metadata packet 700is “Two way peripheral data.” Time 702 indicates the time point at whichthe intended peripheral device should perform a particular actiondesignated by the metadata packet 700. Peripheral ID 703 indicates thatmetadata packet 700 is intended for a particular peripheral device withID “Disney_87”. Peripheral Payload 704 indicates the action to beperformed by the particular peripheral device. The Response Handler 705is a set of instructions intended for the target device, which instructsthe target device how to act after the target device has sent thePeripheral Payload to the desired peripheral device. In their simplestform, these instructions are byte codes, but they can also be higherlevel direct action instructions that are tailored to functionality ofthe target device.

FIG. 8 illustrates an example of possible implementation of actionsincluded in the Response Handler. As shown in FIG. 8, an action can havearguments following it. For example, the possible Two Way PeripheralData Response Handler actions may be “wait N seconds,” “pause playback Nsecond,” “capture activity motion,” “filter for min movement,” “positiveaudio, Good Job”, etc.

Because the format for these actions is target device independent, asame interpreter can be used on multiple target devices, and only arecompile is required for that particular device.

FIG. 9 depicts an example of the data flow of a metadata delivery systemcomprising a peripheral device that can generate response data. As shownin FIG. 9, after the synchronized data 907 is delivered to the targetdevice 901, the target device 901 sends the Peripheral Payload 911 tothe peripheral device 902 and passes the Response Handler 912 to theinterpreter 913. The peripheral device 902 sends data 921 generated byitself back to the target device 901. These generated data 921 are usedas inputs to the interpreter 913 when the actions 914 are processed.Some actions, such as play audio or pause playback, requirecommunication with the native code 915 on the target device 901.

Alternatively, FIG. 10 depicts the data flow of a metadata deliverysystem, which defines standard interfaces for the peripheral device andResponse Handler actions. As shown in FIG. 10, having a standard datainterface 1017 for peripherals cuts down the chance of errors and havinga standard set of actions 1016 simplifies the processing. Similar tothose described above with respect to FIG. 9, after the synchronizeddata 1007 is delivered to the target device 1001, the target device 1001sends the Peripheral Payload 1011 to the peripheral device 1002 andpasses the Response Handler 1012 to the interpreter 1013. The peripheraldevice 1002 sends data 1021 generated by itself back to standardinterface 1017 of the target device 1001. These generated data 1021 areused as inputs to the interpreter 1013 when the standard actions 1014are processed. Some actions require communication with the native code1015 on the target device 1001.

FIG. 11 shows a flow chart for one possible implementation of how toprocess a “Two Way Peripheral Data” packet on a target device. Thetarget device starts the metadata packet processing at step 1101. Themetadata delivery system sends the metadata packet 1102 to the targetdevice. The metadata packet 1102, for example, indicates that this is aTwo Way Peripheral Data type metadata which is intended to peripheraldevice with ID “Fitbit_12” to perform actions indicated by PeripheralPayload at time point “1874472”.

After receiving the metadata packet 1102, in step 1103 a determinationis made regarding whether the type of the metadata packet 1102 is TwoWay Peripheral Data. If the type of the metadata packet 1102 is not TwoWay Peripheral Data, the “NO” line leads to step 1104 where the targetdevice begin to process the metadata as other types of packets. If thetype of the metadata packet 1102 is determined as Two Way PeripheralData, the “YES” line leads to step 1106 where a determination is maderegarding whether it is time to process the packet based on the currenttime 1105. If it is determined that it is not the right time, the “NO”line leads back to step 1106 where determination is made again until thedetermination result is “YES”, then the “YES” line leads to step 1107where the target device sends the Peripheral Payload to all theperipheral devices with ID designated by the Two Way Peripheral Datapacket, e.g., “Fitbit_12”. Then the target device executes ResponseHandler in step 1108, which will be described in detail in FIG. 11. Instep 1109, a determination is made regarding whether there is more datato be processed, and if there is more data, the “YES” line leads back tostep 1103. Otherwise, the “NO” line leads to an end step 1110.

FIG. 12 shows a flow chart for one possible execution of a ResponseHandler on the target device. The target device starts the ResponseHandler executing at step 1201. For example, the metadata packet 1202delivered to the target device indicates that this is a Two WayPeripheral Data type metadata which is intended to peripheral devicewith ID “Fitbit_12” to perform actions indicated by Peripheral Payloadat time point “1874472”, and the Response Handler indicates the actionsto be performed by the target device, such as “wait N second formovement”, “if no movement, play audio ‘pausing video’ wait N seconds”,etc.

After receiving the metadata packet 1202 that includes the ResponseHandler, the interpreter in the target device interprets the action inthe Response Handler in step 1203. After receiving data 1204 from theperipheral device with ID “Fitbit_12”, e.g.,AccelerometerData=“0x123aC237 . . . ”, in step 1205 the interpreterinterprets data in context of Response Handler Action, and the targetdevice execute the needed functions, e.g., pause video, play video, playaudio, etc. In step 1206, a determination is made regarding whetherthere are more actions in Response Handler, and if there are moreactions, the “YES” line leads back to step 1203. Otherwise, the “NO”line leads to an end step 1207. It will be noted that most ResponseHandler actions can be both synchronous and asynchronous.

In another embodiment, the peripheral devices or the target device maybe second screen applications or companion applications, which are meantto complement media played on another screen, such as a primary screen.The peripheral device, such as the toy, may have a built-in targetdevice. The target device and the peripheral may exist in a samephysical object as well as separate, networked objects. Companionapplications are running in proximity to the primary screen. Since boththe primary screen and the secondary screen produce ambient light, toavoid the contrast between these different sources of ambient light thatcreates a distracting or unpleasant viewing environment, the metadatadelivery system may create trigger events within the database using adata map, and send the trigger-points to the companion application tochange the graphical user interface (GUI) of the second screen orcompanion application at specific moments of the media being playback.

FIG. 13 depicts an exemplary data flow of an embodiment of a metadatadelivery system that creates GUI adjustment triggers. The system cangathers information about a media title from a variety of sources andthen linking this information to create an information map of the media,e.g., a film. The triggers can be generated automatically based onautomated audio/video analysis or manually using the data map as aguideline to what GUI changes are most appropriate.

As shown in FIG. 13, a video analyzer and data input application 1301gathers information from three data sources, including media source 1303that provides audio and video stream data 1304, existing media database1305 that provides contextual information about collections of media1306, and data providers 1307 that provides contextual and temporalinformation 1308. An example of data sources that provide contextualdata are described in U.S. patent application Ser. No. 14/588,143entitled Management, Categorization, Contextualizing and Sharing ofMetadata-Based Content and U.S. application Ser. No. 14/587,926,entitled Identifying and Categorizing Contextual Data for Media, andU.S. patent application Ser. No. 14/588,224 entitled Lull Management forContent Delivery, each of which were filed on Dec. 31, 2014 concurrentlyherewith by the same Applicant, OpenTV, Inc., and the entirety each ofwhich is incorporated by reference herein. Further, in variousembodiments manual data can be gathered from paid annotators orcrowd-sourcing applications.

The resulting pool of data can be aggregated and linked through thetimeline of the media and key contextual elements of the media (genre,era, etc. . . . ). The same contextual elements can also be used to linkseveral media titles together and determine useful patterns shared bygroups of film and television programs. These patterns can then be usedto approximate appropriate GUI changes for an unanalyzed media titlebased on similar media titles. Although this data can come from avariety of sources, the data itself can be from automated analysis of atemporal A/V element, contextual data, or other temporal data, examplesof each of which are described below.

FIG. 14 depicts an embodiment of an exemplary data flow for processingaudio and video packets from an elementary stream in a media source1303, to perform automated analysis of a temporal A/V element. Automatedanalysis of an A/V element data can give the system guidelines to mostneeded changes to the GUI. Digitally encoded audio and video can beextracted from an elementary stream 1401 as a frame image (video) 1402or pulse-code modulation (PCM) audio segment 1403. In a standard player,these pieces are rendered and synchronized during audio/video playback.However, these packets can also be fed into various algorithms, such asvideo analyzing algorithms 1404 and audio analyzing algorithms 1405, toanalyze the properties of the audio and video. Some examples of dataderived from these types of analyses are edit detection metrics,color-palette skew (e.g. warm, cool, vibrant, pastel, etc.,) gamma andhistogram distribution of color, luma (i.e. brightness,) facerecognition, song recognition, and peaks in audio amplitude. Forexample, the result of video analyzing algorithms 1404 shows thatpalette skew is vibrant; luminosity is dark; last frame delta is 87%,etc, and the result of audio analyzing algorithms 1405 shows thatamplitude max is 0.71, duration is 123 ms, etc. The results of theseanalyses can be mapped to a timeline using the timestamps that are builtinto the digital video format meant to synchronize audio and videoplayback. One example of this form of timestamps is the presentationtimestamp (PTS) of the MPEG-2 transport stream. The result of thisprocess is a collection of information about the audio and video mappedto the timeline of a specific media title.

Referring back to FIG. 13, existing media databases 1305 storenon-time-related, contextual information about media titles. Using thetitle of the media in question, these databases can provide data such asfilmmakers, genres, release dates, series or serial relationships, andso on.

In various embodiments the system is configured to process contextualdata. FIG. 15 illustrates an example of finding relationships betweenfilms to estimate GUI triggers. Each of these data points can be queriedto find other media titles similar to the original media title. Forexample, a query of the film The Thin Man 1501 could return that thefilm's genres are mystery 1502 and comedy 1503, and the release year1504 of the film is 1934. The film's writer is Dashiell Hammett 1505,and the director is W. S. Van Dyke 1506. The stars in the film includeWilliam Powell 1507 and Myrna Loy 1508. The film has a sequel which isAfter the Thin Man 1509. Further analysis of these attributes of TheThin Man 1501 would reveal several similarities to the film The MalteseFalcon 1510 whose release year 1511 is 1941, since they share the samewriter Dashiell Hammett 1505, same genre mystery 1502 and same generalera attribute, e.g. 7 year apart. It also shows that the director offilm The Maltese Falcon 1510 is John Huston 1512, and the stars includeHumphrey Bogart 1513 and May Astor 1514. This relational informationcould prove important if a system has an AV-Element Analysis of The ThinMan 1501, but not of The Maltese Falcon 1510. In this case it may beviable to use some generalized (non-temporal) GUI modificationsdetermined or defined for The Thin Man 1501 during playback of TheMaltese Falcon 1510.

Referring back to FIG. 13, data providers 1307 may provide othertemporal data, for example contextual and temporal information 1308 forevents that occur in film that are difficult or impossible to determineprogrammatically and automatically as described above. For example thegeographical setting within the narrative space is very difficult todetermine from video analysis of the frame images, especially if thelocation in question only exists in the fictional space of the media.Other examples include era, art direction and props. All of thisinformation can be manually entered by human contributors and attachedto moments along the timeline. For example, a person can note that afilm takes place in a Victorian mansion in rural England during specificmoments. Now the location, era, and artistic styling of the narrativespace can be stored in a database along with the times within the filmduring which this data is valid.

As shown in FIG. 13, the data collected from three data sources may belinked together using the common data markers of the film and thetimeline of the film. That is, from time x to time y in a specific mediatitle, the video analyzer and data input application 1301 performs adigital analysis of the audio and video combined with the temporal dataprovided by human contributors. There is general information about themedia title for estimating unknown data from other related media titles.The aggregated data is generated and the video analyzer & data inputapplication 1301 sends the contextually and temporally synchronizedvisual palettes 1302, i.e., a data map, to database 1309.

FIG. 16 illustrates an example of segments of a data map. Segment 1601,which includes the information of cut 1611, indicates scene number is 3,location of the cut is Lord Busness's Tower, luma of the cut is 126, andcolor distribution of the cut 1611 which is illustrated by histogramsshowing the intensity of Red, Green, and Blue. Similarly, segment 1602,which includes the information of cut 1612, indicates scene number is 4,location of the cut is Western Saloon, luma of the cut is 98, and colordistribution of the cut 1612. Segment 1603, which includes theinformation of cut 1613, indicates scene number is 4, location of thecut is Lego Plane, luma of the cut is 202, and color distribution of thecut 1613. Segment 1604, which includes the information of the cut 1614,indicates scene number is 5, location of the cut is Western Town Saloon,luma of the cut is 102, and color distribution of the cut 1614.

In various embodiments the data map can be used to create trigger eventswithin the database that tell the companion application how and when tochange its GUI at specific moments of the film. These GUI adjustmenttriggers can be created manually with the creator of the trigger usingthe data map as reference. Triggers can also be created automaticallyusing the AV-Element data to guide changes like color, volume, orbrightness.

For example, the GUI adjustments may be simple changes like adjustingthe brightness or volume of the companion application to complement thesource media as it changes throughout playback, or adjustments may becomplicated changes like re-skinning the entire GUI to match the currentera of a time-traveling character as he or she jumps through centurieswithin a film.

FIG. 17 illustrates an example of segments of a data map with GUItriggers. For example, if the GUI adjustment triggers is created basedon the luma of the cut, as shown in FIG. 17, the GUI adjustment 1701 mayadjust the brightness of the companion application to complement the cut1611 whose luma is 126. Similarly, if the GUI adjustment triggers iscreated based on the color distribution of the cut, the GUI adjustment1702 may adjust the color of the companion application to complement thecolor distribution of the cut 1612. If the GUI adjustment triggers iscreated based on the location of the cut, the GUI adjustment 1703 mayadjust the volume of the companion application, for example, to increasethe volume, to complement the cut 1613 whose location is Lego Planewhich, for example, is very noisy.

After the GUI adjustment triggers are created, referring back to FIG.13, the database 1309 sends the contextually and temporally synchronizedGUI instruction 1310 which includes the GUI adjustment triggers to thesecond device 1311. The primary screen 1312 and the second device 1311may be located in the same viewing room 1320. Therefore, the contrastbetween the different ambient light produced by the primary screen andthe second screen can be avoided by sending GUI adjustment triggers toadjust the second screen to complement media played on the primaryscreen.

Accordingly, disclosed is a system which is capable of deliveringsynchronized metadata to a target device for a specific piece of media.The metadata delivery stream includes externally sourced peripheralspecific binary control data packets. The binary data packets are notrendered by the target device, but rather broadcast in a synchronizedfashion any connected peripheral. The binary data packets are only fullyinterpreted by the peripheral itself

Further, disclosed is a system capable of delivering synchronizedmetadata to a media playback target device, such as a set top box ortablet. The target device forwards some of the metadata to connectedintelligent peripherals. In various embodiments, a metadata packet canbe configured for peripherals that are able to generate data and sendthat data back to the target device. The packet can include a payload tobe sent to the peripheral and a response handler, which gets processedon the target device in parallel with the peripheral processing thepayload. The response handler includes a set of target deviceindependent actions to be taken as well as instructions for how tointerpret data generated by the peripheral. In order to process theresponse handler actions and instructions, the target device may beextended with an interpreter.

While the invention has been described and illustrated in connectionwith various embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe scope of the invention, and the invention is thus not to be limitedto the precise details of methodology or construction set forth above,as such variations and modification are intended to be included withinthe scope of the invention. Therefore, the scope of the appended claimsshould not be limited to the description and illustrations of theembodiments contained herein.

1. A metadata delivery system including at least one computer processorand computer readable storage medium including computer code and atleast one storage device, the system comprising: one or more processorsprogrammed at least to deliver, to a target device, metadata packetsrelated to media which includes audio and video data, wherein themetadata packets include textual metadata packets which include timinginformation related to the media, and peripheral metadata packets whichinclude the timing information related to the media, peripheraladdressing information for a plurality of peripherals associated withthe target device, and a peripheral payload including a first set ofactions; render metadata based on the textual metadata packets on afirst screen device associated with the target device; deliver theperipheral payload to the plurality of peripherals, and control theplurality of peripherals to perform the first set of actions based onthe peripheral payload.
 2. The metadata delivery system of claim 1,wherein when the plurality of peripherals generate and send responsedata to the target device, the metadata packets delivered to the targetdevice further include a response handler including a second set ofactions; and wherein the one or more processors are further programmedat least to control the target device to perform the second set ofactions based on the response handler.
 3. The metadata delivery systemof claim 2, wherein the second set of actions are target deviceindependent, and the one or more processors is further programmed atleast to interpret the response handler.
 4. The metadata delivery systemof claim 3, wherein the one or more processors is further programmed atleast to define standard interfaces for the plurality of peripherals andresponse handler.
 5. The metadata delivery system of claim 1, whereinwhen one of the plurality of peripherals is a second screen devicerunning a companion application associated with the target device, themetadata packets being delivered to the target device further includingtrigger-points causing adjustments of a graphical user interface (GUI)of the second screen device at specific time points of the media.
 6. Themetadata delivery system of claim 5, wherein the trigger-points aregenerated based on a data map created by aggregating media informationgathered from a plurality of data sources along timeline of the media.7. The metadata delivery system of claim 6, wherein the plurality ofdata sources include one or more of an automated analysis result of theaudio and video data of the media, contextual information about themedia, and manually entered data from annotators or crowd-sourcingapplications.
 8. The metadata delivery system of claim 5, wherein theadjustments of GUI include adjusting brightness or volume of thecompanion application, or re-skinning the GUI.
 9. The metadata deliverysystem of claim 1, wherein the one or more processors are furtherprogrammed at least to record movements of a first peripheralsynchronized with the media and caused by a user's operations on thefirst peripheral.
 10. The metadata delivery system of claim 9, whereinthe one or more processors is further programmed at least to control asecond peripheral to perform the recorded movements.
 11. The metadatadelivery system of claim 1, wherein when the target device loses syncwith the media, the one or more processors is further programmed atleast to stop delivering the peripheral payload to the plurality ofperipherals until the target device regains sync with the media.
 12. Themetadata delivery system of claim 1, wherein the metadata packets aresourced externally to the metadata delivery system in a just-in-timetransmission before being delivered to the target device.
 13. Themetadata delivery system of claim 1, wherein the plurality ofperipherals and the target device perform a discovery process byexchanging identification data.
 14. A method comprising, in at least onecomputer and a computer readable storage medium including computer code:delivering, to a target device, metadata packets related to media whichincludes audio and video data, wherein the metadata packets includetextual metadata packets which include timing information related to themedia, and peripheral metadata packets which include the timinginformation related to the media, peripheral addressing information fora plurality of peripherals associated with the target device, and aperipheral payload including a first set of actions; rendering metadatabased on the textual metadata packets on a first screen deviceassociated with the target device; delivering the peripheral payload tothe plurality of peripherals, and controlling the plurality ofperipherals to perform the first set of actions based on the peripheralpayload.
 15. The method of claim 14, wherein when the plurality ofperipherals generate and send response data to the target device, themetadata packets delivered to the target device further include aresponse handler including a second set of actions; and method furthercomprises controlling the target device to perform the second set ofactions based on the response handler.
 16. The method of claim 15,wherein the second set of actions are target device independent, and themethod further comprises: interpret the response handler.
 17. The methodclaim 16, wherein the method further comprises defining standardinterfaces for the plurality of peripherals and response handler. 18.The method of claim 14, wherein when one of the plurality of peripheralsis a second screen device running a companion application associatedwith the target device, the metadata packets being delivered to thetarget device further including trigger-points causing adjustments of agraphical user interface (GUI) of the second screen device at specifictime points of the media.
 19. The method of claim 18, wherein thetrigger-points are generated based on a data map created by aggregatingmedia information gathered from a plurality of data sources alongtimeline of the media.
 20. The method of claim 19, wherein the pluralityof data sources include one or more of an automated analysis result ofthe audio and video data of the media, contextual information about themedia, and manually entered data from annotators or crowd-sourcingapplications.